Repository navigation
Fix the 4.11 review findings across detection, guardrails and releases - #187
Merged
Merged
Conversation
release: DataFog Python 4.9.0
Resolve backend through 'auto': the native engine when datafog-core imports, the pure-Python engine otherwise with a single warning. DATAFOG_BACKEND and backend="python" force the fallback; non-regex engines stay on the Python ML path. Serve prose PERSON and ADDRESS from the embedded model in datafog-core, so names and addresses need no spaCy, GLiNER, download or interpreter. Expose the new credential, account and UUID entity types through the alias API and add datafog.native for the native schema. Deprecation warnings now name 4.10.0 as the last supported version with an exact pin, and VERSIONING.md plus docs/retirement.rst make the window and the removal schedule auditable. Nothing is removed in this release.
1.2.0 shipped without the exchange-code fix, so 4.10.0 depends on the release that has it. Contract file follows the engine version.
The corpus encodes Python-engine expectations, so it pins backend=python by default and scores the native engine under DATAFOG_ACCURACY_BACKEND=rust against tests/contracts/native-corpus.json, which records the twelve known divergences with reasons. CI runs both variants; pre-commit gains a ruff per-file-ignores entry for the dynamic re-exports in datafog/native.py.
feat: native engine by default with Python fallback (4.10.0)
Core 1.4.0's text-mode email scanning trims label syntax by default, so `email=alice@example.com` reports `alice@example.com` and quoted values drop the wrapping quote. This moves the reviewed Rust contract file to `rust-1.4.0.json` (no new deviations: every recorded case still matches, verified by running the observation harness against 1.4.0), raises the dependency floor, and resolves the `json-escaped` and `markdown-code-block` native-corpus divergences. `unicode-chinese-name` stays with a corrected reason: its corpus fixture contains literal `?` characters, which are valid local-part characters, so the reviewed expectation remains overlap-only. Also stops the local verification gap this exposed: the two async/optional dependency failures and one accuracy improvement are relative to an unmodified main in the same environment (11 failed there, 10 here).
Merged
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Structured PII now uses the same backend across regex, smart, spaCy, GLiNER, CLI and legacy entrypoints. Lone surrogates preserve source offsets, NER uses real spans without dropping structured findings, TextService covers chunk boundaries, and dense results and fallback patterns avoid the reported quadratic behavior. Guardrails await async results and cover tool arguments, prompt/input shapes and complete buffered streams with request-scoped tokens.
The remaining review changes repair pipeline scan defaults, algorithm selection and optional HMAC, allowlist timeouts, entity validation, Pydantic/anonymizer/Spark behavior, CLI secret output and packaging. Existing plain-hash identities and empty legacy anonymizer filters stay compatible. Scheduled releases qualify without publication; manual runs default to dry runs, reject stale histories and occupied versions, and record validated refs before upload. The branch also brings the stable changes missing from dev into development.
Local validation includes the non-slow suite, installed-wheel smoke, frozen/native contracts, all pre-commit hooks and actionlint. The real spaCy/GLiNER native corpus passed 526 cases with 45 existing expected failures and zero skips; only two exact URL spans are newly permitted. Historical 4.8.1 observations and prior reviewed Core 1.4 fixtures are unchanged. Every Python finding has its resolution and evidence in docs/review-remediation.md.
Core 1.5.0 is the separate unpublished minor-release candidate; public APIs and wire contract 1 stay available. This parent PR intentionally retains the published Core 1.4 minor pin for standalone CI. The dependent release update raises it to ~=1.5.0 and must pass installed published-artifact qualification after Core publication. No publication is performed.